3.2.0 (2024-09-30)

⚠ BREAKING CHANGES

• api: Fail log queries when executed on instant query endpoint (#13421)
• jsonnet: convert read statefulset into deployment for loki-simple-scalable (#13977)
• blooms: Remove bloom compactor component (#13969)
• blooms: Introduce a new block schema (V3) (#14038)
• blooms: Index structured metadata into blooms (#14061)

Features

• ability to ignore cache for volume queries (#13945) (b1dc076)
• add _extracted suffix to detected fields conflicts (#13993) (ab1caea)
• add backoff mechanism to the retention process (#14182) (3136880)
• Add baseline rf1 querier implementation (#13639) (3a99b69)
• Add CLI to inspect RF=1 WAL segments (#13552) (150e653)
• add functions to common.libsonnet for warpstream (#14123) (2bde071)
• Add initial support for a kafka-based ingest path (#13992) (33c26f4)
• add loki_ingester_rf1_segment_age_seconds metric (#13653) (2dfc13b)
• Add metrics for Ingester RF-1 (#13510) (d4179aa)
• Add metrics to WAL Manager (#13490) (bfe97d7)
• add missing cluster label to mixins (#12870) (547ca70)
• add retries for s3 ObjectExists calls (#14062) (73cbbb0)
• Add settings for cpu/mutex/block profiling options (#13278) (f06eabb)
• add structured metadata to the promtail push API (#14153) (66cffcb)
• Added getting started video for ingesting Otel logs (#13226) (5e560f9)
• aggregate byte and count metrics (#13731) (913e9f9)
• api: Fail log queries when executed on instant query endpoint (#13421) (ce71f1c)
• blooms: Add bloom planner and bloom builder to backend target (#13997) (bf60455)
• blooms: Add series & chunks per block metrics (#13721) (55c6499)
• blooms: Index structured metadata into blooms (#14061) (a2fbaa8)
• Bootstrap metastore for wal segments (#13550) (0b47498)
• Calculate the age of a WAL segment (#13637) (4abb5a4)
• chunks-inspect: support structured metadata (#11506) (1834065)
• collect and serve pre-aggregated bytes and counts (#13020) (467eb1b)
• Collect duplicate log line metrics (#13084) (40ee766)
• convert WalSegmentWriter to io.ReadSeeker (#13340) (19c0509)
• detected-labels: remove cardinality filter (#13652) (4f534d7)
• distributors: Use a pool of worker to push to ingesters. (#14245) (f80d68a)
• downsample aggregated metrics (#13449) (2c053ee)
• Drain uses different tokenizer based on log format (#13384) (bc01e6f)
• exclude and from creating new tokens in patterns (#13395) (8c8454b)
• exclude in pattern tokens (#13397) (e612dd3)
• exporter: include boolean values in limit exporter (#13466) (4220737)
• flush not owned streams (#13254) (2ca1ac6)
• helm: ✨ add additional service annotations for components in distributed mode (#14131) (5978f13)
• helm: Add persistence option to memcached on Helm chart (#13619) (ef1df0e)
• helm: Make gateway container port configurable. (#13294) (05176e4)
• helm: Replace bloom compactor with bloom planner and builder (#14003) (08e61ca)
• helm: Support alibabacloud oss in helm chart (#13441) (3ebab6f)
• Ignore empty streams in distributor if all entries fail validation (#13674) (6c4b062)
• Implement GetObjectRange for all storage providers (#13650) (d9c441e)
• implement IsRetryableErr for S3ObjectClient (#14174) (fc90a63)
• Implement owned streams calculation using Partition Ring (#14282) (3c36ba9)
• Implement WAL segment ingestion via Kafka with partition ring (#14043) (d178f4c)
• improve Owned Streams feature observability (#13232) (ce86459)
• improve placeholder replacement of byte sizes (#13508) (ac284ca)
• Ingester RF-1 (#13365) (7f35179)
• Ingester Stream Limit Improvements (#13532) (ec34aaa)
• ingester: Smooth out chunk flush operations (#9994) (82fbb2f)
• instrument failed chunk encoding/decoding (#13684) (5a87ccb)
• Instrument metastore GRPC calls (#13598) (04613b4)
• Instrument rf1 write path with tracing (#13599) (ce88286)
• Introduce a new Object Storage WAL format. (#13253) (1d6f8d5)
• Introduce new ObjectExistsWithSize API to (#14268) (ac422b3)
• Introduce wal segment read path. (#13695) (917053a)
• jsonnet: Allow to name prefix zoned ingesters (#14260) (fac3177)
• kafka: Add Ingestion from Kafka in Ingesters (#14192) (b6e9945)
• kafka: enqueue commit offset only once per batch process (#14278) (beca6f3)
• lambda-promtail; ensure messages to Kinesis are usable by refactoring parsing of KinesisEvent to match parsing of CWEvents + code cleanup (#13098) (dbfb19b)
• lambda-promtail: Adding S3 log parser support for AWS GuardDuty (#13148) (2d92fff)
• lambda-promtail: build lambda with zip file (#13787) (9bf08f7)
• Limit to block ingestion until configured date (#13958) (b5ac6a0)
• loki: add ability to disable AWS S3 dualstack endpoints usage (#13785) (bb257f5)
• loki: Do not enforce max-query-bytes-read and max-querier-bytes-read in limited tripperware (#13406) (47f6ea5)
• max-allowed-line-length: add config to set max-allowed-line-length in pattern ingester (#14070) (0780456)
• mixins / allow bloom dashboards disabling (#14177) (ce2e6d5)
• mixins: Allow hiding useless rows in loki-operational (#13646) (3aa4f22)
• mixins: merge resources dashboards for ssd into one (#13471) (45b8719)
• move detected field logic to query frontend (#14212) (36ace66)
• operator: Add alert for discarded samples (#13512) (5f2a02f)
• operator: Add support for Loki OTLP limits config (#13446) (d02f435)
• operator: Add support for the volume API (#13369) (d451e23)
• operator: Enable leader-election (#13760) (1ba4bff)
• operator: Update Loki operand to v3.1.0 (#13422) (cf5f52d)
• operator: Update Loki operand to v3.1.1 (#14042) (7ae1588)
• Optionally require writes to kafka on Push requests (#14186) (7c78232)
• Pattern ingesters add a limiter for high eviction rate (#13464) (e08b4a7)
• Remove flush loop and queue from Ingester RF-1 (#13538) (8ca03a2)
• remove mutexes from wal.SegmentWriter (#13641) (7ed63ea)
• revert "feat: add functions to common.libsonnet for warpstream" (#14129) (18c27f9)
• RF1 improves replication stategy to support 1 replica (#13469) (790135b)
• rf1: Add query path for the metastore (#13636) (8cb19a2)
• rf1: Store index ref in metastore (#13613) (5f5fd4e)
• structured-metadata-api: add structured metadata to /detected_fields API (#13604) (ce02cc2)
• support ruler sidecar in singleBinary mode (#13572) (684baf7)
• track discarded data by usageTracker (#14081) (c65721e)
• Track when builders are doing work with a gauge (#13570) (0029d46)
• Update doc-validator version (#13558) (a88a0d3)
• upgrade prometheus (#13671) (b88583d)
• Use prefixed WAL storage path in Object Storage (#13377) (973aa2d)
• Use WAL Manager (#13491) (8f1d12f)
• WAL Manager (#13428) (15c8b45)
• WAL Manager configuration options (#13531) (c4405fe)
• wal: Add sizing information to writer and reader. (#13267) (41fbacd)
• wal: Benchmark and improve WAL writes using Reset. (#13272) (debb5f2)

Bug Fixes

• detected_fields: return parsed labels when parsers are passed (#14047) (aa1ac99)
• add a retry middleware to all the stats handlers (#13584) (7232795)
• Add additional validation for timeout while retrieving headers (#14217) (8322e51)
• add logging to empty bloom (#13502) (c263a68)
• add missing flush op timeout (#13679) (62c5c5c)
• Adjust tailer loop criteria so it is actually re-tested (#13906) (dabbfd8)
• aggregated-metrics: correctly create logfmt string (#14124) (63e84b4)
• allow any level for aggregated metrics (#14255) (c001a1d)
• allow rename of structuremetadata labels (#13955) (2d4792a)
• ast left cycular reference result in oom (#13501) (6dd6b65)
• blooms: Cleanup temp blockdir in bloom compactor (#13622) (64215e1)
• blooms: Delete outdated metas during planning (#13363) (11e1976)
• blooms: ensure tokenizer cache is reset between series (#13370) (04bc3a4)
• blooms: Fix eviction of multiple blockcache items (#13573) (c9950e3)
• blooms: Fix panic in bloom gateway (#13303) (66f97b2)
• blooms: Fix panic in initialisation of the bloom planner and builder (#14110) (8307c42)
• blooms: ignores bloom filtering errors in bounded shard query planning (#13285) (ede6941)
• blooms: Improve error wrap to make ignoreNotFound work when fetching blocks (#13656) (bd20171)
• blooms: improves mempool metrics (#13283) (d36e1d5)
• blooms: Minor fixes and improvements for testing in dev (#13341) (d0f56ee)
• blooms: Remove backoff from notify planner (#13506) (e506995)
• blooms: Remove unused arg (#13343) (fcb9b28)
• blooms: Ship chunkrefs in task payload (#13677) (450bbce)
• blooms: skip empty blooms on reads (#13500) (bfa6955)
• blooms: Suppress error from resolving server addresses for blocks (#13385) (3ac2317)
• blooms: Use correct key to populate blockscache at startup (#13624) (2624a4b)
• break: helm: Fix how we set imagePullSecrets for enterprise-gateway and admin-api. (#13761) (3be5a45)
• canary: Reconnect immediately upon tail max duration (#14287) (9267ee3)
• ci: add cleanup step into job dist (#13801) (217f928)
• ci: fixed release-please manifest (#13810) (f253db5)
• cloud-mixin: Attribute OTLP route correctly to write path (#13943) (b91b782)
• config: Copy Alibaba and IBM object storage configuration from common (#14297) (59ff1ec)
• correct _extracted logic in detected fields (#14064) (1b3ba53)
• Dedup pattern tokens on output (#13534) (e23598d)
• deps: update aws-sdk-go-v2 monorepo (#13986) (6f49123)
• deps: update github.com/axiomhq/hyperloglog digest to af9851f (#13806) (67295e0)
• deps: update github.com/c2h5oh/datasize digest to aa82cc1 (#13807) (a93f38c)
• deps: update github.com/docker/go-plugins-helpers digest to 45e2431 (#13808) (e5a3994)
• deps: update github.com/grafana/dskit digest to 7c41a40 (#14277) (f39cdbd)
• deps: update github.com/grafana/dskit digest to 931a021 (#14032) (7c18642)
• deps: update github.com/grafana/jsonparser digest to ea80629 (#13814) (d5718eb)
• deps: update k8s.io/utils digest to 702e33f (#14033) (b7eecc7)
• deps: update module github.com/aliyun/aliyun-oss-go-sdk to v2.2.10+incompatible (#13861) (6f79194)
• deps: update module github.com/azure/go-autorest/autorest/adal to v0.9.24 (#13862) (8041bd2)
• deps: update module github.com/azure/go-autorest/autorest/azure/auth to v0.5.13 (#13863) (71c4421)
• deps: update module github.com/baidubce/bce-sdk-go to v0.9.186 (#13864) (3c0e3e2)
• deps: update module github.com/baidubce/bce-sdk-go to v0.9.187 (#13933) (56af84d)
• deps: update module github.com/baidubce/bce-sdk-go to v0.9.188 (#14000) (79039a2)
• deps: update module github.com/baidubce/bce-sdk-go to v0.9.189 (#14044) (7fb34b4)
• deps: update module github.com/cespare/xxhash/v2 to v2.3.0 (main) (#13615) (cfc7b34)
• deps: update module github.com/datadog/sketches-go to v1.4.6 (#13865) (1f3c467)
• deps: update module github.com/docker/docker to v25.0.5+incompatible [security] (main) (#12279) (960c034)
• deps: update module github.com/docker/docker to v27.1.1+incompatible [security] (main) (#13762) (f8bf3bb)
• deps: update module github.com/docker/docker to v27.1.2+incompatible (#13872) (8ab4c20)
• deps: update module github.com/efficientgo/core to v1.0.0-rc.3 (#14001) (90f7e5f)
• deps: update module github.com/felixge/fgprof to v0.9.4 (#13870) (c68848f)
• deps: update module github.com/fsouza/fake-gcs-server to v1.47.7 (#13935) (d43b2de)
• deps: update module github.com/gogo/googleapis to v1.4.1 (#13871) (6da7eb5)
• deps: update module github.com/gorilla/mux to v1.8.1 (main) (#13618) (19b288e)
• deps: update module github.com/gorilla/websocket to v1.5.3 (#13873) (1eb8342)
• deps: update module github.com/hashicorp/consul/api to v1.29.4 (#14002) (e11b244)
• deps: update module github.com/hashicorp/raft to v1.7.1 (#14005) (e9cec1d)
• deps: update module github.com/ibm/go-sdk-core/v5 to v5.17.4 (#13892) (b6991f2)
• deps: update module github.com/ibm/go-sdk-core/v5 to v5.17.5 (#14045) (677d217)
• deps: update module github.com/ibm/ibm-cos-sdk-go to v1.11.0 (#13893) (9b7e7e9)
• deps: update module github.com/klauspost/pgzip to v1.2.6 (#13874) (fdea7a1)
• deps: update module github.com/mattn/go-ieproxy to v0.0.12 (#13876) (775bf8e)
• deps: update module github.com/minio/minio-go/v7 to v7.0.76 (#14006) (51f9376)
• deps: update module github.com/ncw/swift to v2 (#13951) (246a1df)
• deps: update module github.com/oschwald/geoip2-golang to v1.11.0 (#13934) (3bebba5)
• deps: update module github.com/schollz/progressbar/v3 to v3.14.6 (#13884) (fb9cae4)
• deps: update module github.com/spf13/afero to v1.11.0 (#14060) (bbbd82b)
• deps: update module github.com/tonistiigi/fifo to v1 (#13952) (96b5c79)
• deps: update module github.com/workiva/go-datastructures to v1.1.5 (#13885) (d817aee)
• deps: update module golang.org/x/text to v0.17.0 (main) (#13794) (df61482)
• deps: update module golang.org/x/time to v0.6.0 (#13910) (dff00bd)
• detected fields incorrect type bug (#13515) (f6a94d3)
• detected labels: response when store label values are empty (#13970) (6f99af6)
• detected_fields: always return empty array as null (#14112) (93009d4)
• detected_labels: Add matchers to get labels from store" (#14012) (25234e8)
• do not retain span logger created with index set initialized at query time (#14027) (4e41744)
• downgrade grpc to fix regression (#14065) (8c38d46)
• enable service detection for otlp endoint (#14036) (4f962ef)
• Expand matching for additional variations (#14221) (71d7291)
• fix a bug where AppendRequest with no entries triggers flush (#13672) (8a3ae22)
• Fix HPA ingester typo (#13158) (4ca9785)
• Fix log line for fingerprint not found (#13555) (aeb23bb)
• Fix panic in BloomStore initialisation (#13457) (5f4b8fc)
• Fix panic in ingester.go (#13557) (dbff69a)
• fix retry code to handle grpc status codes. updated newer stats retries to be wrapped with spans (#13592) (d3e1edb)
• fixed typo in ruler URL (#13692) (1476498)
• Fixes pattern pruning stability (#13429) (7c86e65)
• Fixes span name of serializeRounTripper (#13541) (4451d56)
• flaky test blockPlansForGaps (#13743) (37e33d4)
• gateway client: Fix nil pointer dereference panic when using ruler in ring mode (#13436) (304db10)
• Handle block offset exceeding chunk length in memchunk.go (#13661) (d42476a)
• Handle EOF when reading from some obj stores (#13868) (98a15e2)
• helm: Check for rbac.namespaced condition before creating roles (#14201) (3f47f09)
• helm: fix extraObjects (#13107) (b7fcf2b)
• helm: fix imagePullSecrets for statefulset-results-cache (#13051) (8434b2f)
• helm: Fix persistence configuration for Memcached (#14049) (ee6e1cf)
• helm: fixed memcached and provisioner templates (#13788) (1bf9791)
• helm: removed helm test (#13651) (ef03476)
• helm: Update yaml file ./production/helm/loki/Chart.yaml (+1 other) (#13392) (b5b861c)
• helm: Update yaml file ./production/helm/loki/values.yaml (+1 other) (#13426) (fc3904e)
• helm: Various fixes and enhancements for bloom components (#14128) (dc0cbd4)
• Improve execution speed for queries with label filters (#13922) (40f4f14)
• Include whitespaces in extracted tokens (#13738) (7683a79)
• incorrect pod matcher for compactor in mixin when using ssd mode (#12846) (515e13c)
• Init internal server log along with loki's server instance (#13221) (66b8c9b)
• kafka: Fixes partition selection in distributors (#14242) (3f47233)
• kafka: Fixes writer initialization for arm32 (#14115) (4da035b)
• Keep blocks referenced by newer metas (#13614) (784e7d5)
• Lint issues after merge to main (#13326) (7e19cc7)
• log results cache: include pipeline wrapper disabled in cache key (#13328) (221491c)
• log-to-span: timestamp.Time should be called with milliseconds (#14196) (f8d9143)
• loki-mixin: Fix latency panel for Index Gateway (#13629) (f586c00)
• missing dep PartitionRing for Ingester (#14292) (6354ded)
• mixin: Remove pod label from disk usage aggregation (#14180) (5d45c96)
• mixins / loki-resources-overview panel layout (#14178) (8f54ec6)
• mixins: add backend path section in loki-operational for single scalable deployment (#13023) (16881ab)
• mixins: disk space utilization panels with latest KSM versions (#13486) (0ea7431)
• mixin: various latency panels in operational dashboard should have ms unit type instead of seconds (#13260) (f5a9905)
• operator: Allow structured metadata only if V13 schema provided (#13463) (3ac130b)
• operator: Don't overwrite annotations for LokiStack ingress resources (#13708) (f523530)
• operator: Remove duplicate conditions from status (#13497) (527510d)
• operator: Set object storage for delete requests when using retention (#13562) (46de4c1)
• operator: Skip updating annotations for serviceaccounts (#13450) (1b9b111)
• operator: Support v3.1.0 in OpenShift dashboards (#13430) (8279d59)
• operator: Watch for CredentialsRequests on CCOAuthEnv only (#13299) (7fc926e)
• panic when parsing and extracting JSON key values (#13790) (5ef83a7)
• pattern: Fixes latency metric namespace for tee to pattern (#14241) (ae955ed)
• promtail: validate scrape_config job name, do not allow duplicate job names (#13719) (f2d3499)
• Propagate headers/warnings/stats from quantile downstreams (#13881) (a0c7598)
• Propagate query stats from quantile & topk queries (#13831) (78b275b)
• Properly reset wal segment writer (#13468) (6ea83b4)
• protect ruler remote-write overrides map with a mutex when creating new appenders (#13676) (e9a9c60)
• querier address in SingleBinary mode (#13297) (29f1ea9)
• query engine: Include lines with ts equal to end timestamp of the query range when executing range aggregations (#13448) (e0ca67d)
• Read "404" as object not exist (#13901) (3c9c647)
• Read full buffer from storage when fetching a range (#13948) (115fef4)
• record datasample queries are limited query type (#13930) (ae938d0)
• Redo ingester profile tagging (#13239) (32097c8)
• remove limit middleware for detected_labels (#13643) (2642718)
• Remove query size limit for detected fields (#13423) (1fa5127)
• remove retries on the stats handlers because they already retry (#13608) (1008315)
• remove tenant label tagging from profiles to reduce cardinality (#13270) (f897758)
• remove trailing backtick in verify-config for Loki 3.0 (#13640) (498f29a)
• remove usage of unsafe string in label adapter unmarshal (#14216) (758364c)
• Rename mispelled filename (#14237) (cf1d4a3)
• report correct status code for metric and log queries in metrics.go (#12102) (900751c)
• Report PSRL error message correctly (#14187) (a475153)
• Retain original values in logfmt pattern tokenizer (#13535) (5fa9c4b)
• retry: fix retries when using protobuf encoding (#13316) (a457c5d)
• Return empty vector instead of nil for empty evaluator. (#13485) (08615bf)
• sanatize structured metadata at query time (#13983) (3bf7fa9)
• separates directory creation from permission checks (#13248) (1086783)
• sharding: apply offset to both from and through in shard request (#14256) (17c472d)
• sharding: use without() grouping when merging avg_over_time shard results (#12176) (eb8a363)
• skipping label if it contains special symbol (#14068) (55e374e)
• special case the return values from a sharded first/last_over_time query (#13578) (29a37d5)
• split the error and agg metric cases for clarity (#13913) (d2474fa)
• stream ownership check (#13314) (5ae5b31)
• support multi-zone ingesters when converting global to local limits for streams in limiter.go (#13321) (e28c15f)
• try reading chunks which have incorrect offset for blocks (#13720) (7e224d5)
• Update AWS storage timeout error for Go 1.23 behavior (#14226) (a4642b5)
• update fixed limit once streams ownership re-checked (#13231) (7ac19f0)
• Update Loki v3 Helm statefulset-ingester.yaml template (#13118) (5b4e576)
• Use elements match in gapsBetweenTSDBsAndMetas test (#13722) (4cbe2a6)
• Wait for OwnedStreams service in Ingester startup (#14208) (a4aee4f)

Performance Improvements

• Avoid looking up unnecessary TSDB symbols during Volume API (#13960) (7c1a849)
• blooms: always return bloom pages to allocator (#13288) (0cb3ff1)
• blooms: Avoid tiny string allocations for insert cache (#13487) (652ad24)
• blooms: mempool no longer zeroes out buffers unnecessarily (#13282) (eb1cd4c)
• blooms: Remove compression of .tar archived bloom blocks (#14159) (cdf084f)
• Gather aggregate per-line and per-tenant metrics for Drain patterns (#13368) (bf1d6e3)
• Limit tokens per pattern to 128 (#13376) (a1efafd)
• logql: Micro-optimizations for IP filter (#14072) (c5083c7)
• mempool: Replace sync.Mutex with sync.Once (#13293) (61a9854)
• Prune unused pattern branches (#13329) (5ef18cf)
• Re-introduce fixed size memory pool for bloom querier (#13172) (4117b6c)

Miscellaneous Chores

• blooms: Introduce a new block schema (V3) (#14038) (5395daf)
• blooms: Remove bloom compactor component (#13969) (b75eacc)
• jsonnet: convert read statefulset into deployment for loki-simple-scalable (#13977) (1535183)